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Objective 

To  provide  guidance  on  integrating  software  architecture  analysis  and 
evaluation  in  DoD  and  Government  system  acquisitions. 


This  presentation  contains  both  implicit  and 
explicit  guidance.  We  will  use  this  icon  to 
indicate  when  we  are  giving  explicit  guidance. 


The  guidance  is  based  on  real  experience  and  a  series  of  technical 
notes  and  reports  that  were  developed  by  the  Business  and 
Acquisition  Guidelines  (BAG)  group  of  the  Product  Line  Systems 
Program  at  the  SEI.  (See  References.) 
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Key  Definition 

The  software  architecture  of  a  program  or  computing  system  is  the 
structure  or  structures  of  the  system,  which  comprise  software 
elements,  the  externally  visible  properties*  of  those  elements,  and  the 
relationships  among  them. 


*  assumptions  other  elements  can  make  of  a  element,  such  as  its 
provided  services,  performance  characteristics,  fault  handling,  shared 
resource  usage,  and  so  on 


Reference: 

Software  Architecture  in  Practice,  2nd  Edition; 

Bass,  L.;  Clements,  P.;  &  Kazman,  R.  Reading,  MA:  Addison-Wesley  Publishing  Co., 

Spring  2003. 
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Terminology 

Architecture  analysis  refers  to  analyzing  a  system’s  software 
architecture  in  accordance  with  a  prescribed  method. 

Evaluation  is  used  strictly  in  an  acquisition  context — i.e.,  in  reference 
to  performing  an  appraisal  during  source  selection  or  contract 
performance. 

Architecture  analysis  and  evaluation  means  analyzing  an  architecture 
(and  reporting  the  results)  and  evaluating  the  analysis  results  in  strict 
accordance  with  the  technical  evaluation  criteria  of  Section  M  of  the 
RFP  or  the  terms  of  the  governing  contract. 
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Why  Is  Architecture  So  Important? 

Architecture  is  a  common  high-level  communication  vehicle  for  system 
stakeholders  that  is  amenable  to  analysis  and  synthesis. 

Architecture  embodies  the  earliest  set  of  design  decisions  about  a 
system.  These  decisions 

•  are  the  most  profound 

•  are  the  hardest  to  get  right 

•  ripple  through  the  entire  software  development  effort 

•  are  most  costly  to  fix  downstream 

•  are  critical  to  achieving  mission/business  goals 

The  earlier  we  reason  about  tradeoffs,  the  better.  Architecture 
provides  a  powerful  way  to  predict  system  qualities. 
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Software  Versus  System  Acquisition 

“We  buy  systems,  not  software.” 


Promoting  this  message  to  DoD  acquirers  translates  into  “don’t  worry 
about  software.” 

Reality: 

You  should  worry  about  software. 

Software  is  critical  to  systems. 

Software  and  software  architectures  drive  both  functionality 
and  system  quality. 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


PLS  BAG  -  page  8 


( Icimegie  Mellon 

Software  Engineering  Institute 


Relationship  of  System  Requirements  to 
Software  Architecture 
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Being  Proactive  Pays  Off 

Architecture  analysis  and  evaluation  enables  an 
acquisition  program  to 

•  obtain  early  visibility  and  technical  insight  into 

-  system  concept  of  operation 

-  system  and  software  design  decisions  and  tradeoffs 

-  ability  to  achieve  desired  system  quality  attributes 

•  achieve  increased  stakeholder  communication  across  and  within 
government  and  contractor  organizations 

•  identify  and  reduce  risk  early  on — for  new  and  legacy  systems 
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Justification  for  Architecture  Analysis  and 
Evaluation  in  a  System  Acquisition 


i 

i 

i 

i 


a  risk  reduction 
measure 

This  is  consistent  with 
acquisition  reform  principles. 


Symbol  for  using 
architecture  analysis  as  an 

“acquisition  checkpoint” 

to  reduce  risk 
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SEI’s  Architecture  Analysis  Methods 


The  term  “architecture  analysis”  encompasses  both 


ATAM 


SM 


QAW 


Architecture  Tradeoff 


Analysis  MethodSM 


Quality  Attribute 
Workshop 


Need  to 

•  choose  an  analysis  method  that  fits  your  approach 

•  implement  a  compatible  acquisition  infrastructure 


Architecture  Tradeoff  Analysis  Method  and  ATAM  are  service  marks  of  Carnegie  Mellon  University. 
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The  Analysis  Methods 

ATAM 

•  Purpose:  to  assess  the  consequences  of  architectural  decisions  in 
light  of  quality  attribute  requirements  and  business  goals 

-  uncover  risks  created  by  architectural  decisions 

-  surface  design  tradeoffs 

•  Requires  identified  business  goals  and  a  sufficiently  documented 
software  architecture 

QAW 

•  Purpose:  to  help  determine  key  quality  attributes  and  system 
requirements  before  there  is  a  software  architecture  to  which  the 
ATAM  could  be  applied 
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Characteristics  of  ATAM  and  QAW 


ATAM 

•  Uses  extensive  analysis  of  software  attributes  against  quality 
drivers 

•  Involves  broad  range  of  stakeholders 

•  Requires  close  collaboration  with  architecture  development  team 


QAW 


Complementary  offshoot  of  ATAM 
Intended  for  early  stages  of  conceptual  architecture  development 
Can  begin  while  the  software  architecture  is  still  being  crafted 
-  Elements  of  a  system  and  software  architecture  will  suffice. 
Can  be  done  offline  by  developer 
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ATAM  Overview 


Business 

Drivers 

Quality 

Attributes 

Scenarios 

(  Software 
Architecture 

Architectural 

Approaches 

Architectural 

Decisions 

impacts 


Risk  Themes 

distilled 

into 


T  radeoffs 


Sensitivity  Points 


Non-Risks 


Risks 
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ATAM  Benefits 


Provides  visibility  into  the  consequences  of  architectural  decisions  in 
light  of  quality  attribute  requirements 

•  risks  are  explicitly  identified 

•  architecture  sensitivity  points  are  determined 

•  tradeoffs  are  made  more  explicit 

Increases  communication  among  stakeholders 

Provides  documented  basis  for  making  architectural  decisions 


The  results  are  improved  software  architectures. 
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QAW  Process 
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QAW  Process  -  In  DoD  Environment 


Acquirer’s  Responsibility 


Supplier’s  Responsibility 
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QAW  Benefits 


Clarifies  business  drivers  and  quality  attribute  requirements 

Allows  generation  of  scenarios  and  test  cases  before  there  is  a  software 
architecture 

Results  in  improved  architecture  documentation 
Allows  risks  to  be  identified  early  in  the  life  cycle 
Provides  documented  basis  for  architectural  decisions 
Increases  communication  among  stakeholders 

The  results  are  improved  conceptual  architectures. 
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Conditions  for  using  ATAM  or  QAW 

ATAM 

•  Stakeholders  want  to  evaluate  the  software  architecture. 

•  Business  drivers  and  system  quality  attributes  are  well 
understood. 

•  Software  architecture  is  suitably  documented. 

•  Architecture  development  team  is  available  for  engagement. 

QAW 

•  An  architecture  analysis  is  desirable  to  reduce  risk. 

•  Business  drivers  and  system  quality  attributes  need  refinement. 

•  There  is  only  a  conceptual  architecture  and  limited  architectural 
documentation. 

•  A  very  flexible  analysis  method  is  needed  to  allow  for  early 
intervention. 
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Applying  Architecture  Analysis  and 
Evaluation  in  a  System  Acquisition 

In  competitive  acquisitions,  architecture  analysis  and  evaluation  can 
be  used  to  help  manage  the  solicitation  process,  including  source 
selections. 

After  contract  award,  architecture  analysis  can  be  used  to  help 
manage  the  contract  performance  process,  including  contractor 
performance  and  product  evaluations. 

How  to  use  architecture  analysis  and  evaluation  most  effectively 
depends  on  your  objectives  and  system  acquisition  strategy. 
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Architecture  Analysis  and  Evaluation  in 
Solicitation 

Architecture  analysis  and  evaluation  can  be  used  as  a  discriminator 
in  the  source  selection  process.  Options  include  evaluating  each 
offeror’s 

•  related  experience  in  architecture  development  and  analysis 

•  ability  to  adequately  plan  an  architecture  analysis  using  a 
prescribed  method 

•  actual  progress  against  an  approved  analysis  plan 

•  architecture  analysis  results 

•  demonstrated  ability  to  take  suitable  remedial  action  and  mitigate 
discovered  risks 
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Architecture  Analysis  and  Evaluation  in 
Contract  Performance 

During  system  development,  architecture  analysis  can  be  used  to 

•  select  an  architecture  among  several  candidate  architectures 

•  assist  in  decision  whether  to  upgrade  or  rebuild  legacy  system 

•  assist  in  architecture  refinement  once  an  architecture  has  been 
chosen 

-  during  progressive  stages  of  software  development 

-  during  evaluation  of  new  system  builds 

During  system  sustainment,  architecture  analysis  can  be  used  to 
support  system  upgrades  and  reengineering. 

During  system  development  and  sustainment,  architecture  analysis 
and  evaluation  can  play  a  role  in  incentive  awards  to  the  degree 
specified  in  the  contract. 
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Guidance:  Basic  Principles 


Remember  that 


One  size  doesn’t  fit  all. 


-  The  approach  for  integrating 

architecture  analysis  and  evaluation  in  a  system  acquisition 
has  to  be  adapted  to  the  situation. 

•  Success  will  depend  on  carefully  planning  a  coherent  approach 
and  integrating  it  with  the  system  acquisition  strategy. 

•  Analysis  aspects  must  faithfully  adhere  to  the  specified  architecture 
analysis  method. 

•  Evaluation  aspects  must  comply  with  the  FAR*  and  be  specified  up 
front  in  the  RFP  and  contractual  requirements. 


Federal  Acquisition  Regulations 
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Example  System  Acquisition  Strategy 


RFP  #  1 


Contract 

Awards 


RFP  #2 


Contract 

Award 


Contractor  A  Performance 


Open 
Competition 


Source 

Selection 


\  \  \ 


Evaluate 

Deliverables 


Selection  i  System  Development 


S/S 


Contractor  B  Performance 


> 


Open  Solicitation  j 

Competitive  Fly-Off 

|  Main  Contract  Performance 

- H 

and  Initial  Down  Select  1 

and  Final  Down  Select 

~~ n  *■ 
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Potential  Application  of  Architecture  Analysis 
and  Evaluation 


Part  of  Oral 

Technical  Presentations 
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How  Do  You  Incorporate  Architecture  Analysis 
and  Evaluation  in  an  Acquisition? 


Db yal  op  nn  Itiisgtciisd 
Approc  ich 

that  includes 

si  Proscribed  Soi:  at 
Eiverns  nnc\  AmhiatB 


The  purpose  of  these  events  and  artifacts  is  to 
create  a  suitable  acquisition  infrastructure. 
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Guidance:  Getting  Started 


To  integrate  architecture  analysis 
and  evaluation  you  will  need  to 


•  understand  the  policies  and  constraints  retimoiDg-its 
that  govern  the  acquisition 

•  understand  the  system  acquisition  strategy  and  proposed  technical 
evaluation  criteria  for  the  RFP/contract 

•  have  a  good  working  knowledge  of  the  architecture  analysis 
method  to  be  applied 

•  understand  how  the  system’s  technical  requirements  apply  to  the 
architecture  analysis  and  evaluation 

•  strongly  consider  adopting  a  competitive  fly-off  strategy  to  reduce 
risk  by  allowing  more  detailed  analysis  (suppliers)  and  evaluation 
(acquirers) 
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Developing  an  Integrated  Approach  i 

•  Establish  your  objectives  for  incorporating  architecture  analysis 
and  evaluation. 

•  Review  the  proposed  acquisition  strategy. 

•  Conduct  a  brainstorming  session  with  stakeholders  to  explore 
how  architecture  analysis  and  evaluation  can  best  be  applied  in 
source  selection  and  contract  performance. 

•  Select  the  most  beneficial  course  of  action  that  is  compatible  with 
the  selected  architecture  analysis  method. 

•  Identify  what  events  must  take  place  and  what  artifacts  must  be 
produced  in  each  phase  of  the  acquisition. 
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Developing  an  Integrated  Approach  2 

•  Define  a  compatible  set  of  roles  and  responsibilities  for  both  the 
acquirer  and  supplier. 

•  Identify  the  specific  tasks  that  the  acquirer  and  supplier  must 
perform  in  each  phase  of  the  acquisition  and  the  artifacts  that  they 
must  develop  and  deliver. 

•  Have  stakeholders  review  and  critique  the  proposed  approach. 

•  Adjust  the  approach  or  acquisition  strategy  as  necessary  and 
obtain  the  approval  of  the  contracting  officer. 

•  Develop  corresponding  RFP/contract  language  to  implement 
steps  4  through  9  in  an  effective  manner. 
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Acquirer’s  Key  Events  and  Artifacts 

In  planning  the  approach,  give  due  consideration  to 

•  specifying  the  business/mission  drivers 

•  specifying  the  system  quality  attributes  and  architecture  test  cases 
and  architecture  documentation 

•  conducting  a  tutoria  on  the  architecture  analysis  method  for 
prospective  offerors  before  issuing  the  RFP 

•  holding  a  kickoff  meeting  after  contract  award  to  describe 

-  government  expectations  for  the  supplier’s  architecture 
analysis 

-  rules  of  engagement  for  conducting  the  analysis  and 
evaluation 

•  providing  formal  feedback  on  the  analysis  results 

•  having  an  equitable  basis  for  evaluating  the  final  outcome 
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Supplier’s  Key  Events  and  Artifacts 

In  planning  the  approach,  give  due  consideration  to 

•  requiring  suppliers  to  prepare  a  preliminary  architecture  analysis 
plan  to  demonstrate  their  understanding  of  the  analysis  method 

•  requiring  suppliers  to  conduct  a  dry  run  architecture  analysis  to 
assess  their  level  of  preparedness 

•  requiring  the  active  participation  of  key  stakeholders,  including  the 
supplier’s  software  architect 

•  incorporating  an  equitable  means  for  suppliers  to  deliver  and 
present  their  analysis  results  and  respond  to  official  feedback  from 
the  acquirer 
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Software  Architecture  Guidance 


DoD  typically  mandates  use  of 
the  C4ISR  Framework. 

This  often  translates  to  “C4ISR  is  sufficient  to  address 
all  architectural  concerns.” 

Reality: 

The  C4ISR  Framework  provides  important  context 
but  is  not  a  software  architecture. 
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Guidance:  In  Developing 
Your  Specific  Approach 


Accommodate  all  aspects  of  the  architecture  analysis  method  in  the 
most  practical,  straightforward  manner. 

Ensure  that  the  RFP  development  team  is  included  in  all  the 
acquisition  technical  planning  and  decision-making  activities. 

Include  the  contracting  officer  in  the  planning  process. 

Ensure  that  companies  contributing  to  RFP/contract  preparation  will 
exclude  themselves  from  bidding  to  avoid  a  potential  protest. 

If  adopting  a  competitive  fly-off  approach,  use  a  Call  for  Improvement 
(CFI)  instead  of  issuing  another  RFP. 
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Example  System  Acquisition  Strategy 
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Integrating  a  QAW  (Parti) 
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Integrating  a  QAW  (Part  2) 


Initial 

Down  Select 

Competitive 

Fly-Off 

Final 

Down  Select 

System 

Implementation 

J _ 

W 

i  n 

Proposal 

Evaluation 

and 

Contract 

Awards 


Refined 

Architecture 

Analysis 

Plan 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


PLS  BAG  -  page  40 


( Iciruegie  Mellon 

Software  Engineering  Institute 


Integrating  a  QAW  (Part  2  with  EN) 


Evaluation  Notices 


4 


ENs 


At  the  request  of  the  acquirer’s  evaluation 
team,  the  contracting  officer  issues 
evaluation  notices  (ENs)  to  inform 
contractors  of 

•  items  needing  clarification 

•  deficiencies  requiring  resolution 

REQUESTS  FOR  CLARIFICATIONS  cite 
conditions  (e.g.,  ambiguities,  anomalies,  and 
issues)  requiring  further  explanation  or 
correction  by  the  contractor. 

DEFICIENCIES  cite  conditions  that 
represent  potential  risks  in  achieving  the 
desired  system  quality  attributes  (including 
failure  to  meet  expected  architecture  test 
case  response  in  a  QAW)  or  conditions  th< 
do  not  satisfy  the  system  requirements. 
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Integrating  a  QAW  (Part  2  Continued) 
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Integrating  ATAM  (Part  3) 
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Implementing  the  Architecture  Analysis  and 
Evaluation  Approach 

What  happens  during  solicitation  and  contract  performance 
critically  depends  on  what  is  included  in  the  RFP  and  the 

resulting  contract. 

Appropriate  RFP/contract  language  must  be  developed  to  make 
architecture  analysis  and  evaluation  an  integral  part  of  evaluating 
proposals  as  well  as  evaluating  system  aspects. 

Only  the  RFP  and  contract  language  can  give  the  government  the 
means  to  manage  the  suitability  of  the  software  architecture. 
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Incorporating  architecture  analysis  requires  developing 
appropriate  language  for  the  following  sections: 


Description,  SOW  ,  Performance  Specification 


Statement  Of  Objectives  (SOO) 


Special  Contract  Requirements 


include 


Contract  Deliverables  Requirements  List 


Instructions,  Conditions,  and 
Notices  to  Offerors 


Evaluation  Factors  for  Award 


Statement  of  Work 
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RFP  Section  C 


Performance  Specification* 

Contains  two  sections  of  interest: 

•  Section  1  covers  the  technical  requirements  for  the  acquisition. 

•  Section  2  describes  the  mechanisms  for  confirming  that 
requirements  have  been  met. 

Section  1  specifies  system  quality  requirements  from  which  software 
architecture  requirements  (runtime  and  non-runtime)  are  derived. 
They  must  be  stated  in  terms  of 

•  definition  of  system  quality  attributes 

•  specification  of  acceptable  values 

•  definition  of  scenarios  and  test  cases 

•  specification  of  other  requirements  (e.g.,  C4ISRJ 

Section  2  includes  a  description  of  the  architecture  analysis  method. 


*  System  Specification  or  Technical  Requirements  Document 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


PLS  BAG  -  page  47 


CarnegieMellon 

Software  Engineering  Institute 


RFP  Section  H 


Section  H:  Special  Contract  Requirements 

Section  H  will  specify 

•  the  architecture  analysis  requirements 

•  the  artifacts  that  are  to  be  developed 

•  rules  of  engagement  (ROE) 

Section  H  will  describe 

•  when  the  architecture  analysis  should  be  performed 

•  how  the  analysis  should  be  conducted 

•  contractor  and  government  roles  and  responsibilities 

•  how  the  results  will  be  used 
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RFP  Section  H 


Typical  Rules  of  Engagement  (ROE) 

•  Offerors  will  submit  architecture  analysis  plan  with  proposal 
based  on  government-prescribed  architecture  analysis  method. 

•  Contracting  officer  will  notify  selected  suppliers  of  any 
deficiencies  in  their  analysis  plan. 

•  Supplier  will  conduct  architecture  analysis  and  present  results  in 
accordance  with  its  approved  plan. 

•  Contracting  officer  will  issue  evaluation  notices  (ENs)  to  suppliers 
following  presentation  of  analysis  results. 

•  Suppliers  are  required  to  respond  to  all  ENs. 

•  Contract  takes  precedence  over  the  architecture  analysis  plan. 

•  Analysis  results  are  evaluated  in  accordance  with  contract  or 
technical  evaluation  criteria  of  Section  M  of  RFP. 
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RFP  Section  J 


There  are  no  DoD  Data  Item  Descriptions  (DIDs)  specific  to 
architecture. 

However,  some  DoD  organizations  have  tailored  the  following  DIDs 
from  DoD-2167A  that  contain  architecture  information  to  meet  their 
acquisition  needs: 

•  System/Subsystem  Design  Document  (SSDD) 

•  Software  Design  Document  (SDD) 

Under  acquisition  reform,  many  contractors  will  describe  the  specific 
artifacts  and  documentation  they  will  prepare  per  their  own  internal 
processes. 
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RFP  Section  L 


Section  L  describes  what  offerors  should  address  in  their  proposal;  that 
is,  which  requirements  in  the  RFP  need  a  response. 

Offerors  prepare  multiple  volumes. 


Volume  3 


Volume  5  + 


Technical 

Past  Performance 

Pre-Award  Demonstration 
Plan  /  Procedures 
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Specific  Section  L  Volumes 

In  the  ^echnical  volume,  you  ask  offerors  to  describe  their  approach 
for  implementing  and  analyzing  architecture  requirements. 

In  the  Past  Performance  volume,  you  ask  offerors  to  describe 
previous  work  on  software  architecture  development  and  architecture 
evaluation. 

In  the  Pre-Award  Demonstration  volume,  you  give  offerors 
requirements  for  demonstrating  the  capability  of  their  software 
architecture. 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


PLS  BAG  -  page  52 


Carnegie  Mellon  _ 

Software  Engineering  Institute  ■SBWH 

RFP  Section  M:  Evaluation  Criteria 


RFP  Section  M 


Section  M  tells  offerors  how  their  proposals  will  be  evaluated. 

It  should  specify 

•  the  ranking  of  evaluation  factors 

•  how  architecture  analysis  is  made  part  of  the  specific 
evaluation  factors 

•  the  evaluation  criteria  forjudging  the  proposal  submission  and 
how  results  of  the  architecture  analysis  will  be  included  in  the 
technical  evaluation 
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Guidance:  Some  Precautions 

To  avoid  potential  problems 

•  Understand  the  policies  and  criteria  that  the  acquisition  review 
board  will  use  as  part  of  the  RFP  review  and  approval  process. 

•  Make  sure  the  architecture  analysis  and  evaluation  approach  has 
the  “buy  in”  of  the  contracting  officer. 

•  Firm  up  the  acquisition  strategy  and  technical  evaluation  criteria 
early  in  the  acquisition  planning  phase. 

•  Resist  management  pressures  to  arbitrarily  compress  the 
acquisition  schedule,  including  RFP  preparation. 

•  Include  personnel  with  domain  and  architecture  expertise  on  the 
RFP  development  team. 
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Guidance:  RFP  Preparation 


It  takes  time  to  carefully  draft  the  right  wording  for  architecture 
analysis  and  evaluation  and  integrate  it  into  an  RFP  in  a  manner 
that  is  compatible  with  the  acquisition  strategy  and  addresses  all 
the  key  issues. 


If  you  rush  your  preparation  of  the  RFP  you  will  likely  increase 
downstream  risk  and  not  reap  all  the  benefits  of  an  architecture¬ 
centric  acquisition  approach. 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


PLS  BAG 


Cam^bMdlon 

Software  Engineering  Institute 

Topics 


Introduction 

Motivation  for  architecture  analysis 
Architecture  analysis  methods 

Integrating  architecture  analysis  and  evaluation  into  a  system 
acquisition 

Example  of  an  integrated  approach 
RFP/contract  language  considerations 

Summary 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


PLS  BAG  -  page  56 


CarnegieMellon 

Software  Engineering  Institute 


Summary  i 


Our  acquisition  example  illustrates  the  use  of  software  architecture 
analysis  and  evaluation  in  source  selection  and  contract 
performance. 

A  prescribed  set  of  events  and  artifacts  can  be  used  to  integrate 
architecture  analysis  and  evaluation  in  a  system  acquisition. 

This  approach  works  with  any  system  and  software  development 
methodology. 

RFP  and  contract  language  alone  can  give  the  acquirer  the 
capability  to  manage  the  system  quality  of  the  architecture. 
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Summary  2 


Architecture  analysis  and  evaluation  provide  an  effective  means  for 

•  mitigating  risks  in  a  system  acquisition 

•  evaluating  the  achievement  of  system  quality  attributes  that  are 
important  to  the  program 

•  changing  customary  program  assumptions  about  software 
oversight  that  underlie  system  acquisitions 
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